Vehicle and trailer wheel path collision detection and alert

ABSTRACT

A method including detecting an obstacle location in response to an image, determining the obstacle location in response to a range measurement received from a range sensor, predicting a tow vehicle path in response to a tow vehicle steering angle and a tow vehicle location, predicting, by a processor, a trailer wheel path in response to the tow vehicle path, a vehicle dimension and a trailer dimension, and generating a warning control signal in response to an intersection of the obstacle location and the trailer wheel path.

INTRODUCTION

The present disclosure relates generally to a system for providingtrailer guidance and assistance within a motor vehicle. Morespecifically, aspects of the present disclosure relate to systems,methods and devices for determining a trailer path during traileringoperations, detecting objects within the determined trailer path andproviding a warning to a vehicle system algorithm or a vehicle operator.

Personal motor vehicles have long been fitted with equipment to tow atrailer behind the vehicle. These trailers may include camping trailers,boat trailers or utility trailers. These trailers are typicallyphysically connected to a vehicle trailer hitch mechanically fixed to atowing vehicle frame. The trailers are connected to the trailer hitchwith a hitch arrangement coupled to a frame of the trailer. In addition,electrical connections are provided between the towing vehicle and thetrailer to supply electrical power to trailer lighting and brakingsystems, as well as to couple electronic signals to control theapplication of the trailer brakes and illumination of various trailerlights, such as brake lights, turn signal lights, reverse lights and thelike.

Difficulty arises in that during a trailering operation, a trailer pathmay be outside that of the tow vehicle path. During cornering, thetrailer wheels often follow a path inside the wheels of the towingvehicle and trailer wheel contact may occur with curbs if the towvehicle does not make a wide enough turn. This contact may cause damageto the trailer, the trailer wheels, and/or the curb. In addition, otherparts of the trailer may contact objects that the towing vehicle hassuccessfully avoided. It would be desirable to provide a means forproviding feedback to a tow vehicle and/or a tow vehicle operator whileovercoming the aforementioned problems.

SUMMARY

Disclosed herein are vehicle sensor methods and systems and relatedcontrol logic for provisioning vehicle systems, methods for making andmethods for operating such systems, and motor vehicles equipped withonboard control systems. By way of example, and not limitation, thereare presented various embodiments of systems for the object detectionand contact event prediction for a towing vehicle and a trailer.

In accordance with an aspect of the present disclosure, a methodincluding detecting an obstacle location in response to an image,determining the obstacle location in response to a range measurementreceived from a range sensor, predicting a tow vehicle path in responseto a tow vehicle steering angle and a tow vehicle location, predicting,by a processor, a trailer wheel path in response to the tow vehiclepath, a vehicle dimension and a trailer dimension, and generating awarning control signal in response to an intersection of the obstaclelocation and the trailer wheel path.

In accordance with another aspect of the present disclosure, furtherincluding displaying an indicator of the obstacle location and anindicator of the trailer wheel path on a vehicle cabin display.

In accordance with another aspect of the present disclosure, wherein theimage is captured by a camera mounted to the tow vehicle, wherein thecamera has a forward field of view from the tow vehicle.

In accordance with another aspect of the present disclosure, where theobstacle location is confirmed in response to a depth measurement from aside view sensor mounted to the tow vehicle.

In accordance with another aspect of the present disclosure, wherein thevehicle dimension is a distance from a vehicle rear axle to a trailerhitch point and the trailer dimension is at least one of a distance froma trailer wheel axle to a trailer hitch point and a trailer track width.

In accordance with another aspect of the present disclosure, wherein theobstacle location is confirmed in response to a depth measurement froman ultrasonic sensor mounted to the tow vehicle.

In accordance with another aspect of the present disclosure, wherein thetrailer wheel path is predicted in response to a trailer hitcharticulation angle.

In accordance with another aspect of the present disclosure, wherein thetow vehicle path is predicted in response to at least one of anacceleration detected by an inertial measurement unit affixed to a towvehicle and a displacement detected by a wheel encoder affixed to thetow vehicle.

In accordance with another aspect of the present disclosure, wherein theimage is captured in response to a trailer system activation and adetection of motion of a tow vehicle.

In accordance with another aspect of the present disclosure, furtherincluding amending a map data in response to the obstacle location andthe tow vehicle location and wherein the trailer wheel path isdetermined in response to the map data.

In accordance with another aspect of the present disclosure, wherein thewarning control signal is generated in response to at least one of atime to collision and a distance between the obstacle location and thetrailer wheel path.

In accordance with another aspect of the present disclosure, anapparatus including a camera configured for capturing an image of afield of view including an object, a distance sensor configured todetermine a distance to the object, a processor configured to determinean object location relative to a tow vehicle location in response to theimage, confirming the location of the object in response to the distanceto the object, predicting a tow vehicle path in response to a towvehicle steering angle and the tow vehicle location, predicting atrailer wheel path in response to the tow vehicle path, vehicledimension and a trailer dimension, generating a warning control signalin response to an intersection of the object location and the trailerwheel path, and displaying a warning to a vehicle operator, by a userinterface, in response to the warning control signal.

In accordance with another aspect of the present disclosure, wherein theobstacle is a curb.

In accordance with another aspect of the present disclosure, wherein thelocation sensor is a global positioning sensor.

In accordance with another aspect of the present disclosure, wherein thecamera is a stereo camera for capturing a plurality of stereo images andwherein the object location is determined in response to the pluralityof stereo images.

In accordance with another aspect of the present disclosure, wherein thedistance sensor is a side view ultrasonic sensor.

In accordance with another aspect of the present disclosure, furtherincluding a memory for storing a map data and wherein the processor isfurther operative to amend the map data in response to the objectlocation and the tow vehicle location and wherein the trailer wheel pathis determined in response to the map data.

In accordance with another aspect of the present disclosure, the imageis captured in response to a trailer system activation and a detectionof a tow vehicle motion.

In accordance with another aspect of the present disclosure, wherein thevehicle dimension is a distance from a vehicle rear axle to a trailerhitch point and the trailer dimension is at least one of a distance froma trailer wheel axle to a trailer hitch point and a trailer track width.

In accordance with another aspect of the present disclosure, a trailerpath hazard detection system including a trailer interface for detectinga connection to a trailer, a forward view camera for capturing a firstimage including a curb hazard, a side view camera for capturing a secondimage including the curb hazard, a range sensor for determining adistance to the curb hazard, a processor for estimating a location ofthe curb hazard in response to the first image, the second imagerelative to a tow vehicle location, confirming the location of the curbhazard in response to the distance to the curb hazard, predicting a towvehicle path in response to a tow vehicle steering angle and the towvehicle location, predicting a trailer wheel path in response to the towvehicle path, a vehicle dimension and a trailer dimension, andgenerating a warning control signal in response to an intersection ofthe obstacle location and the trailer wheel path, and a display fordisplaying the second image and overlaying a first indicator of the curblocation and a second indicator of the trailer wheel path over thesecond image.

In accordance with another aspect of the present disclosure, furtherincluding a speaker to generating an audible alert in response to thewarning control signal.

BRIEF DESCRIPTION OF THE DRAWINGS

The exemplary embodiments will hereinafter be described in conjunctionwith the following drawing figures, wherein like numerals denote likeelements, and wherein:

FIG. 1 shows an exemplary environment for use of the trailer path hazarddetection system according to an embodiment of the present disclosure;

FIG. 2 shows a block diagram illustrating a system for implementing thetrailer path hazard detection system according to an exemplaryembodiment of the present disclosure;

FIG. 3 shows a flow chart illustrating an exemplary method forcontrolling the trailer path hazard detection system according to anexemplary embodiment of the present disclosure;

FIG. 4 shows a user interface display for providing driver feedback fromtrailer path hazard detection system according to an exemplaryembodiment of the present disclosure;

FIG. 5 shows a block diagram illustrating a system for implementing thetrailer path hazard detection system according to an exemplaryembodiment of the present disclosure; and

FIG. 6 shows another flow chart illustrating an exemplary method forcontrolling the trailer path hazard detection system according to anexemplary embodiment of the present disclosure.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and isnot intended to limit the application and uses. Furthermore, there is nointention to be bound by any expressed or implied theory presented inthe preceding technical field, background, brief summary or thefollowing detailed description. As used herein, the term module refersto an application specific integrated circuit (ASIC), an electroniccircuit, a processor (shared, dedicated, or group) and memory thatexecutes one or more software or firmware programs, a combinationallogic circuit, and/or other suitable components that provide thedescribed functionality.

Turning now to FIG. 1 , an environment 100 for use of the trailer pathhazard detection system according to an exemplary embodiment of thepresent disclosure is shown. The exemplary environment 100 depicts atowing vehicle 105, a trailer 110. The towing vehicle 105 may beequipped with optical imaging sensors, for example a front view camerahaving a camera field of view (FOV) 130 and a right side view camerahaving a right side FOV 131 and a left side camera having a left sideFOV 132, and range imaging sensors, for example, a front left ultrasonicsensor having a front left ultrasonic detection zone 135, a front rightultrasonic sensor having a front right ultrasonic detection zone 137, arear left ultrasonic sensor having a rear left ultrasonic detection zone139 and a rear right ultrasonic sensor having a rear right ultrasonicdetection zone 138.

It is desirable to vehicle operators to keep vehicles and trailers frombeing scratched or damaged. One source of damage to vehicles andtrailers wheels is damage from curbs and parking blocks primarily causedby contact of the wheel rim with the curb while turning. The exemplarysystem is configured to perceive curb hazards and estimate a vehiclewheel path 120 with relation to curb and other object hazards that canoccur and provide guidance to the driver to help prevent collisions. Theexemplary system is configured to perceive curb hazards and estimate atrailer wheel path 125 with relation to curb and other object hazardsthat can occur and provide guidance to the driver to help preventcollisions during trailering operations. The exemplary system may employa fusion of vehicle perception sensors, including optical image sensorssuch as cameras, and range imaging sensors such as ultrasonic sensors,lidar, or short range radar, and other vehicle kinematic and dynamicdata along with algorithms to detect, localize, track, and alert curbhazards and other hazardous objects inside the turning radius of thetrailer.

In some exemplary embodiments, a tow vehicle 105 is configured with asystem to perceive curbs and other hazards proximate to the tow vehicle105 and trailer 110 when performing a trailering operation. Theexemplary system determines if the vehicle wheels or trailer wheels orother parts of the trailer may contact a curb 115 or other hazard. Ifso, the exemplary system then provides guidance to a vehicle operator ora vehicle control system to help prevent these contact events andsubsequent damage to the vehicle 105 or trailer 110.

The exemplary system may incorporate a number of perception sensorsincluding optical imaging sensors for example front and side cameras,and range imaging sensors for example ultrasonic sensors or short-rangeradars, along with a processor, an algorithm, and customer interface. Insome exemplary embodiments, the tow vehicle 105 may be equipped withultrasonic sensors on the front corner and rear corner of the towvehicle 105 used to detect and localize a curb and/or other objects nextto the tow vehicle 105. The tow vehicle 105 is further equipped with oneor more forward view cameras having a forward field of view 130 and leftand right side view cameras having a left side field of view 132 and aright side field of view 131.

In some exemplary embodiments, the system may be configured to perform asensor fusion of the detections of each of the sensors to generate alocal area map referenced to the tow vehicle 105. As the tow vehicle 105moves, the local area map is updated with newly acquired information andexisting information is confirmed or updated in response to subsequentsensor data. Combining data from several sources using multi sensor datafusion algorithms exploits the data redundancy to reduce object locationuncertainty. In some exemplary embodiments, the sensor data may be fusedusing Bayesian fusion algorithms with Kalman filtering.

The system may further perform an estimation model to detect andlocalize a curb or objects inside the curb by fusing two set ofcomplementary on-board vehicle sensors, including optical imagingsensors such as surrounding cameras, and range imaging sensors such asultrasonic sensors. The estimation model may use a camera-basedalgorithm to capture and map front and side views from the tow vehicle105 and then fuses side ultrasonic sensors with the front view and sideview cameras using a model-based or non-model-based Bayesian filtering.In some exemplary embodiments, the curb hazards or obstacles that weredetected using the fusion of cameras, and ultrasonic sensors, may becombined to generate a map of the area around the tow vehicle 105.

The system may be further configured to monitor tow vehicle and trailertelemetry, such as tow vehicle steering angle and velocity. In responseto the telemetry, the system is then configured to predict the towvehicle and trailer odometry in response to the telemetry. This towvehicle and trailer odometry maps the predicted tow vehicle wheel path120 and/or predicted trailer wheel path 125. This predicted trailerwheel path 125 may be compared to detected obstacles and curbs on thelocal map. If a contact event is predicted in response to the predictedtrailer wheel path 125, an alert may be transmitted to the vehicleoperator via a user interface, such as an audible alert, warning light,seat vibration, or a graphic on an in-vehicle cabin display.

A landmark estimation model may be further employed to enhance theestimation model through the local odometry algorithm that fusesmultiple on-board sensors such as GPS, IMU, wheel pulses, steeringangles. The exemplary algorithm may use local odometry to spatiotemporalstamp the landmarks, and as a result generate a local map of therecently seen landmarks. The algorithm may then utilize akinematics/dynamics model of the tow vehicle 105 with/without thetrailer 110 to estimate the projected path of the tow vehicle 105 ortrailer wheels and to assess that whether that path would collide withthe detected landmarks in the local map. The path guideline is annotatedon the viewing system for the feature to interact with the driver. Thepath guideline may adapt and change with changes in the steeringdirection and hitch articulation angle.

The exemplary system may further include a user interface for providingfeedback to a vehicle operator, such as audible chimes, displays, andhaptic feedback when a potential hazard is detected. For example,guidelines may be displayed to a vehicle operator where guideline colormay represent likelihood of a potential collision between the trailerwheels and curb based on the distance of the wheels to the curb as wellas the time to collision of the wheels to the curb.

Turning now to FIG. 2 , a block diagram illustrating a system 200 forimplementing a trailer path hazard detection system according to anexemplary embodiment of the present disclosure is shown. The exemplarysystem 200 may be located in a towing vehicle and include a processor240, a left side range imaging sensor 221, a right side range imagingsensor 223, a left side optical sensor 224, a right side optical sensor225, a camera 222, a global positioning system (GPS) sensor 245, aninertial measurement unit (IMU) sensor 235, a vehicle controller 270, auser interface 260, a memory 265 and a trailer interface 255.

The camera 222 may be a front facing camera installed on a forwardportion of the towing vehicle, such as a front facia or behind a rearview mirror. The camera 222 may include one or more image sensors forcapturing a forward facing field of view from the front of the towingvehicle. In some exemplary embodiments images captured by the camera222, the left side optical sensor 224 and/or the right side opticalsensor 225 may be fused to generate a combined image of an areaproximate to the tow vehicle.

If more than one image sensor is used, a stereo image may be generated.Image processing algorithms may be performed on the stereo image toestimate depth information in the forward field of view. One or moreimages captured by the camera 222 may then be coupled to the processor240. In addition, the exemplary system may include one or more sideoptical sensors, such as a left side optical sensor 224, a right sideoptical sensor 225, used to capture images and/or determine depthinformation for left and right sides of the vehicle and/or trailer.

The left side range imaging sensor 221 and the right side range imagingsensor 223 may be range imaging sensors and/or depth sensors configuredto transmit a pulse, receive a reflection of the pulse and estimate adistance to a vertical surface of an object, such as a curb or otherobstacle, in response to the propagation time of the transmitted pulse.The left side range imaging sensor 221 and the right side range imagingsensor 223 may have fields of view orthogonal to the centerline of thetow vehicle and may be oriented such that the field of view cover anexpected location of a curb or other objects proximate to the side ofthe tow vehicle. In some exemplary embodiments, the left side rangeimaging sensor 221 and the right side range imaging sensor 223 may beultrasonic sensors configured to transmit ultrasonic audio pulse. Thedistance to a curb or object may be estimated in response to thepropagation time of the ultrasonic audio pulse. Alternatively, the leftside range imaging sensor 221 and the right side range imaging sensor223 may be lidar sensor, radar, or other range sensing device.

The GPS 245 is configured to receive a plurality of time stampedsatellite signals including the location data of a transmittingsatellite. The GPS sensor 245 then uses this information to determine aprecise location of the GPS sensor 245. The processor 240 may beoperative to receive the location data from the GPS sensor 245 and/orstore this location data to the memory 265. The memory 265 may beoperative to store map data for use by the processor 240.

The IMU 235 is a device used to report a specific force of a body, suchas the angular or linear acceleration of the body to which it isattached. The IMU 235 may measure angular or linear acceleration and maybe used to determine a lateral acceleration, a longitudinalacceleration, a yaw rate and pitch of a vehicle. The IMU 235 is mountedwithin the tow vehicle and is operative to generate a control signalindicative of the measured specific forces and to couple this controlsignal to the processor 240.

The user interface 260 may be operative to provide a human machineinterface between the vehicle control system and the driver. The userinterface 260 may be a touch sensitive display screen, a display, one ormore buttons, dials, switches, etc., and or one or more light emittingdiodes, audio generation devices, such as a speaker, or otherindicators. The user interface 260 is operative to receive user requestssuch as a request for activating a vehicle system, such as settings foran ADAS operation. In an exemplary embodiment, the user interface 260may be operative to receive an activation of a trailering mode in a towvehicle in response to a user input. The user interface 260 may befurther configured to display user alerts if a vehicle or trailercontact event is predicted. The user interface 260 may display apredicted tow vehicle path and/or a predicted trailer path on a displaywithin the tow vehicle cabin. The user interface 260 may be configuredto receive trailer dimension and/or configuration information inresponse to a user input or may initiate a trailer dimension estimationalgorithm in response to a user input.

In some exemplary embodiments, the user interface 260 may provideaudible and/or visual warnings to a driver which are intensified inresponse to time to collision and/or distance to collision. For example,as a projected trailer wheel path passes within a threshold distance toa curb or other object, the color of a displayed trailer wheel pathindicator may change from green to yellow, or from yellow to red. Anaudible alert may be generated by the user interface 260 in response tothe projected trailer wheel path passing within a threshold distance toa curb or other object. An amplitude or frequency of an audible alertmay be altered in response to a changed in distance between a projectedwheel path and an obstacle such as a curb.

The vehicle controller 270 may be operative to control the vehicleoperations in response to a control algorithm or the like. In anexemplary embodiment, the vehicle controller 270 may be operative togenerate vehicle control signals for coupling to a throttle controller,a brake controller, and a steering controller or the like for control ofthe vehicle in response to an advanced driver assistance system (ADAS)algorithm or in response to a vehicle operator input via a steeringwheel, vehicle pedals, and the like. The vehicle controller 270 mayfurther be operative to generate system control signals for coupling tothe processor 240, indicative of such information as speed of thevehicle, acceleration, vehicle motion cycles, vehicle operator inputstate, trailer etc.

The trailer interface 255 may be a module within the tow vehicleconfigured to receive and transmit control signals and/or data to thetrailer. The trailer interface 255 may be electrically coupled to atrailer connector socket. The socket may be a 7 pin electrical socketwith each of the pins separately electrically coupled to the trailerinterface 255. The socket is configured for receiving a plug that iselectrically coupled to a wiring system of the trailer with anelectrical cord long enough to stay affixed to the socket on the towingvehicle while accommodating the movement of the trailer during towingoperations. In some exemplary embodiments, the trailer interface 255 maybe configured to detect the insertion of the plug into the socket. Thisdetection may be made in response to a change in capacitance or voltageon one or more of the electrical connections to the socket. For example,the trailer interface 255 may detect a change in voltage on the groundpin of the socket. Alternatively, the insertion may be detected inresponse to a change of state of a physical switch, such as a buttonswitch being depressed by the plug as the plug is inserted into thesocket.

The processor 240 is configured to perform the trailer path hazarddetection algorithm. In response to the image data received from thecamera 222 and depth map information received from each of the left siderange imaging sensor 221 and the right side range imaging sensor 223,the processor 240 is first configured to generate a local area map ofobjects and roadway features, such as curbs, proximate to the towvehicle. The processor 240 may augment this local area map with map datastored in the memory 265 or map data received from other sources. Thelocal area map is continuously updated as image data and depthinformation is received. The location of the tow vehicle and the trailerare localized within the local area map such that distances between thedetected obstacles, the tow vehicle position and the trailer positionmay be determined.

The processor 240 is further configured to receive vehicle telemetrydata, such as vehicle speed, steering angle, braking information and thelike, from the vehicle controller 270. The processor 240 may furtherreceive acceleration data from the IMU 235. In response to the telemetrydata and the acceleration data, the processor 240 is configured toestimate an anticipated tow vehicle path. In response to the anticipatedtow vehicle path, hitch articulation angle, tow vehicle geometry andtrailer geometry and measurements, the processor 240 next predicts ananticipated trailer path. The anticipated tow vehicle path and theanticipated trailer path are compared with the map data and possiblecontact events with the tow vehicle, the trailer, or the trailer wheelsare predicted. If a possible contact event is predicted, an alertcontrol signal is generated and coupled to the user interface 260 and/orthe vehicle controller 270. In some exemplary embodiments, the possiblecontact event may be based on the distance of the wheels to the curb aswell as the time to collision of the wheels to the curb. The vehiclecontroller 270 may provide the alert control signals as information toan ADAS algorithm for assisted or autonomous vehicle operation.

In some exemplary embodiments, the processor 240 may first receive animage and/or image data from the camera 222 depicting the forward fieldof view from the tow vehicle. The processor 240 may then perform edgedetection techniques, or other image processing techniques, to estimatethe location of curbs proximate to the tow vehicle. The processor 240may subsequently receive depth data from the left side range imagingsensor 221 and the right side range imaging sensor 223 generated inresponse to a pulse reflection from one or more vertical surfaces andconfirm the location of the curbs. This confirmed information is storedas a map data in the memory 265 or other accessible memory. Theprocessor 240 may then predict a trailer path in response to trailermeasurements and tow vehicle telemetry. In response to the predictedtrailer path, the processor 240 may next predict any possible contactevents between the tow vehicle, tow vehicle wheels, and/or trailerwheels and the curb. If a contact event is predicted, a warning signalis coupled from the processor 240 to the user interface 260 in order toalert a vehicle operator to the possible contact event.

In some exemplary embodiments, a top view of the tow vehicle and thetrailer may be generated from various cameras around the tow vehicleand/or the trailer. A tow vehicle image and a trailer image may beoverlaid on the top view as well as indicators of the predicted towvehicle path and the predicted trailer path. Curbs, roadway indicators,and other objects may be highlighted or indicated on the top down viewwith additional overlays. Predicted contact points may be indicated onthe top down view. For example, possible contact points may behighlighted in a color, such as red, be indicated by an icon, such as anexplosion icon, or may be otherwise indicated to the vehicle operator.

Turning now to FIG. 3 , a flow chart illustrating an exemplary method300 for controlling a trailer path hazard detection system according toan exemplary embodiment of the present disclosure is shown.

The method is first configured to receive 310 an image from a cameramounted to the tow vehicle. The camera may be a wide angle camera havinga forward field of view from the tow vehicle, or may be a wide anglecamera having a side field of view from the tow vehicle. In someexemplary embodiments, the camera may include a plurality of imagesensors capable of capturing a stereo image wherein depth informationmay be determined for objects within the stereo image. The image, or aseries of images captured at regular time intervals, may be then coupledto an image processor. In some exemplary embodiments, the processor maybe configured to detect horizontal edges or lines within the image whichmay be indicative of a curb.

The method is next operative to detect 315 objects within the capturedimage or plurality of images. The method may use sequential imagescaptured at different times to estimate distances to objects using theparallax within the images caused by movement of the tow vehicle. Insome exemplary embodiments, the imaged may be converted to grayscale, aninverse perspective mapping performed, followed by image smoothing.Canny edge detection may be used to detect edges within the image, suchas curbs, light posts, etc. A Hough transform may also be employed toextract features from the images to detect instances of edges or othershapes.

The method is next operative to receive 320 depth information from theside view sensors having a side field of view near the front of thetowing vehicle. These side view sensors may include ultrasonic sensors,cameras, lidar, radar, or the like. In some exemplary embodiments, theside view sensors are configured to generate depth maps of theirrespective fields of view and to couple these depth maps to theprocessor. The depths may be indicative of a distance from the side viewsensor to a vertical surface. In some exemplary embodiments, depthinformation is coupled to the processor, such as two dimensional depthinformation that is captured at regular time intervals with the movementof the tow vehicle. For example, the field of view may be a vertical fanshape returning a plurality of depths at varying elevations with asingle azimuth. The movement of the tow vehicle enables a variation inthe azimuth measurements used to generate the depth map.

The method is next operative to confirm 325 object locations detectedfrom the image using the received depth information from the side viewsensors. For example, a horizontal edge detected in the image may becorrelated with a vertical edge detected in response to the depthinformation to estimate a location of a curb. If the object location isnot confirmed, the method returns to receive 310 the next image. If thelocation is confirmed, the method then updates 330 a map informationstored in a memory.

The method is configured to update 330 the map with object locations inresponse to a landmark estimation model that fuses the front viewcaptured by the camera and the depth information captured by the sideview sensors using Bayesian filtering. The stored map coverage may becropped in response to the tow vehicle position. For example, the methodmay store map information for a predetermined radius around the towvehicle, discarding object information once the object falls out of thepredetermined radius.

The method is next configured to receive 335 telemetry information fromthe tow vehicle controller or other tow vehicle sensors. The telemetryinformation may include vehicle velocity, location, speed, steeringangle, brake application level, various accelerations detected by avehicle IMU or the like. The landmark estimation model may be furtherenhanced through the local odometry algorithm that fuses multipleon-board sensors such as GPS, IMU, wheel pulses, steering angles usingBayesian filtering.

The method next predicts 340 a tow vehicle path in response to thereceived telemetry information. The tow vehicle path is predicted usingthe local map and a kinematics/dynamics model of the tow vehicle toestimate the projected path of the vehicle. The method next uses thepredicted tow path, the trailer dimensions and, in some exemplaryembodiments, the hitch articulation angle, to predict 345 the trailerpath. Specifically, the method may predict the trailer wheel paths.

The method next predicts 350 possible contact events in response to themap data and the predicted trailer path. In some exemplary embodiments,the trailer path hazard detection algorithm may assess whether thetrailer wheel path may intersect with the detected landmarks in thelocal map. If no intersection is predicted, indicating that a contactevent will not occur, the method is configured to return to receiving310 the next image.

If a contact event is predicted, the method is next configured togenerate 355 an alert to provide to the vehicle operator or a vehiclecontrol system performing an ADAS algorithm. For an alert provided tothe vehicle operator, a projected trailer path guideline may beannotated on a graphical user interface of a vehicle cabin display. Insome exemplary embodiments, the path guideline may be adaptive andchanging location with respect to an indicated obstacle with change inthe steering wheel and hitch articulation angle. For example, a curb maybe indicated on the display and the projected trailer wheel path mayalso be projected. As the projected trailer wheel path approaches thecurb location, the indicated projected trailer wheel path gets closer tothe curb indicator and the color of the indicated projected trailerwheel path may change color to highlight the increased likelihood of acontact event. An audible alert, haptic alert, or other vehicle operatorwarning may be generated when a distance to an intersection between theprojected trailer wheel path and the obstacle reaches a thresholddistance. After the alert is generated, the method returns to receiving310 a subsequent image.

In some exemplary embodiments, the method utilizes the vehicle odometrymodel to create a spatiotemporal map of the recently seen landmarks, byfusing sensors included but not limited to cameras, ultrasonics (and/orshort range radars), vehicle dynamics and kinematics sensors and modelincluded but not limited to steering wheel angle, GPS, IMU, wheelencoder. The method fuses the spatiotemporal map of the landmarks aswell as the projected/predictive trajectory of the vehicle/trailerwheels developed to assess whether the vehicle wheels are about tocollide with these landmarks, based on distance of the wheels to thelandmarks or time to collision of the wheels to the landmarks or acombination of both.

The method may determine the projected or predictive trajectory of thevehicle/trailer wheels by utilizing the vehicle/trailer model including,but not limited to, dynamics or kinematics model as well as the vehicleon-board sensors including but not limited to the steering wheel angleand IMU, along with the hitch articulation angle between the vehicle andtrailer and other vehicle and trailer dimensions. The vehicle operatoruser interface and/or feedback system may utilize the assessment andprovide feedback to the driver that may include for example audiblechimes, displays, and haptic feedback when a potential hazard isdetected. Examples include, but are not limited to, guideline colorrepresents potential collision between the trailing wheels and curbbased on the distance of the wheels to the curb as well as the time tocollision of the wheels to the curb.

Turning now to FIG. 4 , an exemplary user interface display forproviding driver feedback from trailer path hazard detection systemaccording to an exemplary embodiment of the present disclosure is shown.The first image 400 is indicative of a side view from the tow vehiclewhere the tow vehicle is beginning to execute a right turn around acorner during a towing operation. The detected curb is highlighted bythe system by overlaying a first indicator 405 over the detected curb. Atrailer guideline 410 is also overlaid on the user interface image toidentify to the driver the projected trailer wheel path. The firstindicator 405 and the trailer guideline 410 may be depicted in differentcolors on the user interface image. In some exemplary embodiments, asthe turn progresses and where the projected trailer wheel path isapproaching the curb, the color of the trailer guideline may changecolors to communicate the increased risk of curb collision to thevehicle operator.

The second image 430 is indicative of a turn where the projected trailerwheel path is close to contacting the curb. In some exemplaryembodiments, the curb indicator 415 and the trailer guideline 420 areshow as nearly overlapping on the user interface image. In someexemplary embodiments, the color of the trailer guideline may againchange color to communicate the imminent curb collision to the vehicleoperation. In addition, additional alerts may be provided to the vehicleoperation, such as a haptic alert, audible alert, flashing lightemitting diodes, or the like, indicative of the imminent curb collision.

Turning now to FIG. 5 , a block diagram of a trailer path hazarddetection system 500 according to an exemplary embodiment of the presentdisclosure is shown. The exemplary system may include a camera 510, adistance sensor 515, a processor 520, a location sensor 525, a memory535 and a user interface 530.

The camera 510 may be configured for capturing an image of a field ofview including an object. In some exemplary embodiments, the object is acurb. The obstacle may be another obstacle, such as a traffic sign,safety bollard, light post, utility pole, or the like, located close toan edge of the roadway. In some exemplary embodiments, the camera may bea stereo camera for capturing a plurality of stereo images and whereinthe object location is determined in response to the plurality of stereoimages. The image may be captured in response to a trailer systemactivation and a detection of a tow vehicle motion.

The distance sensor 515 is configured for determining a distance to theobject. The distance sensor 515 may have a side facing field of view andmay determine the distance to the object at a different time than whenthe image of the object was captured with the camera. These detectionsat different times may be correlated using the tow vehicle locations atthe different times to confirm the object location and generate a mapdata in response to the object location. In some exemplary embodiments,the distance sensor 515 may be a side view ultrasonic sensor.

The location sensor 525 is configured for determining a tow vehiclelocation. The location sensor 525 may be a global positioning sensor andmay determined the vehicle location in response to data transmitted froma plurality of satellites. Alternatively, the location sensor may usevehicle to infrastructure (V2I) signals from local infrastructure todetermine a tow vehicle location.

The processor 520 is configured for detecting an object location inresponse to the image and the tow vehicle location. The processor 520may use image processing techniques such as edge detection to detect theobject within the image. The processor 520 may further estimate adistance to the object in response to one or more images and the towvehicle location.

The processor 520 may be further operative for predicting a tow vehiclepath in response to a tow vehicle steering angle and the tow vehiclelocation. The tow vehicle path may be localized to a map data stored inthe memory 535 and added to the map data. The processor 520 may thenpredict a trailer wheel path in response to the tow vehicle path and atrailer dimension. In some embodiments, the trailer dimension is adistance from a trailer wheel to a trailer hitch point. Alternatively,the trailer dimension may include a plurality of dimensions such astrailer track width, distance from the hitch point to the trailer axle,trailer width, trailer height, trailer length, hitch mechanism length,and hitch articulation angle. The processor may be further configuredfor generating a warning control signal in response to an intersectionof the object location and the trailer wheel path.

The user interface 530 may be configured for displaying a warning inresponse to the warning control signal. The user interface 530 may be awarning light, a speaker for playing an audible alarm, or a hapticdevice. The user interface may be a display within a vehicle cabin fordisplaying one or more images including the curb, for overlying ahighlight on the curb, and for displaying an indication of the trailerwheel path.

The exemplary system may further include a memory 535 for storing a mapdata and wherein the processor is further operative to amend the mapdata in response to the object location and the tow vehicle location andwherein the trailer wheel path is determined in response to the mapdata.

In some exemplary embodiments, the trailer path hazard detection system500 may include a trailer interface for detecting a connection to atrailer, a forward view camera for capturing a first image including acurb, a side view camera for capturing a second image including thecurb, a processor for determining a curb location in response to thefirst image, the second image and a tow vehicle location, predicting atow vehicle path in response to a tow vehicle steering angle and the towvehicle location, predicting a trailer wheel path in response to the towvehicle path and a trailer dimension, and generating a warning controlsignal in response to an intersection of the obstacle location and thetrailer wheel path, and a display for displaying the second image andoverlaying a first indicator of the curb location and a second indicatorof the trailer wheel path over the second image. The trailer path hazarddetection system 500 may further include a speaker to generating anaudible alert in response to the warning control signal.

Turning now to FIG. 6 , a flow chart illustrating an exemplary method600 for implementing a trailer path hazard detection system according toan exemplary embodiment of the present disclosure is shown. Theexemplary method may be performed by a processor within a tow vehicle.The processor may be communicatively coupled to sensors, controllers,and interfaces within the tow vehicle. The processor may further becommunicatively coupled to a user interface, such as a vehicle cabindisplay, a vehicle infotainment system or an application running on asmartphone or the like.

The exemplary method may detect 610 an obstacle location in response toan image where the image is captured by a camera mounted to the towvehicle. The camera may have a forward field of view from the towvehicle. In some exemplary embodiments, the obstacle location may beconfirmed in response to a depth measurement from a side view sensormounted to the tow vehicle. The obstacle location may be confirmed inresponse to a depth measurement from an ultrasonic sensor mounted to thetow vehicle. The image may be captured in response to a trailer systemactivation and a detection of motion of a tow vehicle.

The method may predict 620 a tow vehicle path in response to a towvehicle steering angle and a tow vehicle location. In addition, the towvehicle path may be predicted in response to vehicle telemetry, mapdata, and location sensor data. For example, the tow vehicle path may bepredicted in response to an acceleration detected by an inertialmeasurement unit affixed to a tow vehicle. In some embodiments, the towvehicle path may be predicted in response to a displacement detected bya wheel encoder affixed to a tow vehicle.

The method may next predict 630 a trailer wheel path in response to thetow vehicle path and vehicle and trailer dimensions. The vehicledimension may be the distance from the vehicle rear axle to the trailerhitch point. The trailer dimension may be a distance from a trailerwheel axle to a trailer hitch point and a trailer track width. Thetrailer wheel path may be predicted in response to a trailer hitcharticulation angle.

The method may then compare the predicted trailer wheel path and theobstacle location to determine if a contact event may occur. A contactevent may occur if there is a predicted intersection of the obstaclelocation and the trailer wheel path. The method may generate 640 awarning control signal in response to the predicted contact event. Insome exemplary embodiments, the method may be operative for displaying650 an indicator of the obstacle location and an indicator of thetrailer wheel path on a vehicle cabin display. A display color of thetrailer wheel path indicator may change in response to a currentdistance between the trailer wheel and the obstacle. The method mayfurther be operative for amending 660 a map data in response to theobstacle location and the tow vehicle location and wherein the trailerwheel path is determined in response to the map data

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or exemplary embodiments are only examples, and arenot intended to limit the scope, applicability, or configuration of thedisclosure in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the exemplary embodiment or exemplary embodiments. Itshould be understood that various changes can be made in the functionand arrangement of elements without departing from the scope of thedisclosure as set forth in the appended claims and the legal equivalentsthereof.

What is claimed is:
 1. A method comprising: detecting an obstaclelocation in response to an image; determining the obstacle location inresponse to a range measurement received from a range sensor; predictinga tow vehicle path in response to a tow vehicle steering angle and a towvehicle location; predicting, by a processor, a trailer wheel path inresponse to the tow vehicle path, a vehicle dimension and a trailerdimension; displaying an indicator of the trailer wheel path on avehicle cabin display wherein a color of the indicator of the trailerwheel path changes in response to a minimum distance between the trailerwheel path and the obstacle location; and generating a warning controlsignal in response to an intersection of the obstacle location and thetrailer wheel path.
 2. The method of claim 1, further includingdisplaying an indicator of the obstacle location on the vehicle cabindisplay.
 3. The method of claim 1, wherein the image is captured by acamera mounted to a tow vehicle, wherein the camera has a forward fieldof view from the tow vehicle.
 4. The method of claim 1, where theobstacle location is confirmed in response to a depth measurement from aside view range sensor mounted to a tow vehicle.
 5. The method of claim1, wherein the vehicle dimension is a distance from a vehicle rear axleto a trailer hitch point and the trailer dimension is at least one of adistance from a trailer wheel axle to the trailer hitch point and atrailer track width.
 6. The method of claim 1 where the obstaclelocation is confirmed in response to a depth measurement from anultrasonic sensor mounted to a tow vehicle.
 7. The method of claim 1,wherein the trailer wheel path is predicted in response to a trailerhitch articulation angle.
 8. The method of claim 1, wherein the towvehicle path is predicted in response to a displacement detected by awheel encoder affixed to a tow vehicle.
 9. The method of claim 1 whereinthe image is captured in response to a trailer system activation and adetection of motion of a tow vehicle.
 10. The method of claim 1, whereinthe warning control signal is generated in response to at least one of atime to collision and a distance between the obstacle location and thetrailer wheel path.
 11. An apparatus comprising: a camera configured forcapturing an image of a field of view including an object; a distancesensor configured to determine a distance to the object; a processorconfigured to determine an object location relative to a tow vehiclelocation in response to the image, confirming the location of the objectin response to the distance to the object, predicting a tow vehicle pathin response to a tow vehicle steering angle and the tow vehiclelocation, predicting a trailer wheel path in response to the tow vehiclepath, a vehicle dimension and a trailer dimension; generating a warningcontrol signal in response to an intersection of the object location andthe trailer wheel path; and displaying on a vehicle cabin display, awarning to a vehicle operator, by a user interface, in response to thewarning control signal and an indicator of the trailer wheel pathwherein a color of the indicator of the trailer wheel path changes inresponse to a minimum distance between the trailer wheel path and theobject location.
 12. The apparatus of claim 11, wherein the object is acurb.
 13. The apparatus of claim 11, wherein the tow vehicle location isdetermined in response to a global positioning sensor.
 14. The apparatusof claim 11, wherein the camera is a stereo camera for capturing aplurality of stereo images and wherein the object location is determinedin response to the plurality of stereo images.
 15. The apparatus ofclaim 11, wherein the distance sensor is a side view ultrasonic sensor.16. The apparatus of claim 11, further including a memory for storing amap data and wherein the processor is further operative to amend the mapdata in response to the object location and the tow vehicle location andwherein the trailer wheel path is determined in response to the mapdata.
 17. The apparatus of claim 11, the image is captured in responseto a trailer system activation and a detection of a tow vehicle motion.18. The apparatus of claim 11, wherein the vehicle dimension is adistance from a vehicle rear axle to a trailer hitch point and thetrailer dimension is at least one of a distance from a trailer wheelaxle to the trailer hitch point and a trailer track width.
 19. A trailerpath hazard detection system comprising: a trailer interface fordetecting a connection to a trailer; a forward view camera for capturinga first image including a curb hazard; a side view camera for capturinga second image including the curb hazard; a range sensor for determininga distance to the curb hazard; a processor for estimating a location ofthe curb hazard in response to the first image, the second imagerelative to a tow vehicle location, confirming the location of the curbhazard in response to the distance to the curb hazard, predicting a towvehicle path in response to a tow vehicle steering angle and the towvehicle location, predicting a trailer wheel path in response to the towvehicle path, a vehicle dimension and a trailer dimension, andgenerating a warning control signal in response to an intersection ofthe curb hazard and the trailer wheel path; and a display for displayingthe second image and overlaying a first indicator of the curb hazard anda second indicator of the trailer wheel path over the second imagewherein a color of the second indicator of the trailer wheel pathchanges in response to a minimum distance between the trailer wheel pathand the location of the curb hazard.
 20. The trailer path hazarddetection system of claim 19 further including a speaker to generatingan audible alert in response to the warning control signal.